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, Evaluation of the Vanguard Programs ^ 

A New Approach to Assesisment of 
Programs for the Gifted and Talented 

This paper is a .^status or field report on the development of an 
evaluation design appropriate for assessment of a program for gifted 
and talented students in a large metropolitan area. Certain charac- 
teristics both of the program arid ^f the student body made conven- 
tional pre-and posttest evaluation designs inappropriate; in addi- 
tioh/ program admiriistrators wanted information about the prbgraS not 
usually collected in standard evaluatibri. This papet details (aj the 
genesis of the program? (bj its unique fctiaracteristics; (c) a 
discussion of methods for evaluatibri appropriate to programs of this 
type; the evaluation design i:nd data cbllectibri techniques 

utilized iri the evaluation; and (ej a description of the prbgr^s 
made so far in implement at ibri bf the evaluation design. Our 

goal is to develop an evaiulation model which (a) irivblves both 

_ _ * 

pibgram' and evaluatibri personnel in planning and implementation of 

> \ _._ __ _ 

the ev,aluation/ and (b) is apprbpriate for assessment of programs for 

exceptional children i 

Ge'nesis bf the Prbgram 

In 1972/ the Houston Independent School District began its first 

program for educationally gifted children. Called Vanguard, it 

u . ' _ . _ _ _ _ _ . . . . ■ . _ 

emphasized ihterdiscipliriary arid multi- level instruction. It 

r _ _____ 

was specifically designed to assist in the develbpmerit bf creative and 
gifted students' .through individualized teaching methods which allowed 
opporttmitttes for the student to advance as rapidly as they were able. 

b 



in 1975/ the Vanguard program was incorporated into HlSti's Magnet 

School programs Establishment of the Magnet School Program was one of 

many approaches which HISD had used in order to comply with Supreme 

Court decisions regarding desegregation of public education. Other 

options such as fr^om of choice > school pairing, and integration of 

school faculties had rfet achieved the desired results; they had, in 

fact , proven to accelerate the degree of racial isolation in the 

1 _ _ 

Houston schools. The first phase of the Magnet School program was 

was implemented formally in the i975«1976 academic year; it involved 

the restoration of equidistant school attendance zones and the 

establishment of approximately 40 Magnet programs. Some of these, 

such as the Vanguard prbgrim, were developed from existing HISD 

programs; others were entirely new. The eatablishement of the 

Van^ard program as a component of the Magnet Schools program brought 

a new ^imensibh to recruitment arid management of^ the gifted and 

talented program. 

The mairi thrust of Magnet School philosophy is to achieve iritegira- 

\^ ______ 

tion of public schools through provisiori df high quality educational 

programs. Thus, the race of students and the ethriic composition of 

both sending and receiving schools are important. The Magnet Schools 

Program, is intended both to reduce the number of one-race schools in 

the district arid to increase the percentage of students attending 

integrated schools. This is the primary goal of the Magnet Schools 



^Ma^et School Proi^t Evaluation, 5th ftnnuil Report 1979-1980. 
Research Evaluation ^4nd Accreditation, Houston Independent School 
District 
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Program^ and to that eria, each component has both a * set number of 
students who can be accommodated and an ethnic ratio reflecting 
district-wide ethnic ratios which must be met* While some leeway 
is allbwed to individual programs ^ in general students with appropriate 
ethnicity aire recruited to fill designated vacancies. At the present 




timer the required ethnic ratio for Magnet Schools is 35% White and 



ether and 65% Black and Hispanic. 

Traditional lyr students have been identified for gifted and 
talented programs primarily oh the basis of high scores on tests of 
intelligence and academfc achievement. Because white middle class and 
upper SES children tend to do better on such tests than do minority 
children r programs for the academically talented have tended to ex- 
elude Black and Mexican-fimertcan children. Since children will display 
their giftedness in culturally approved patterns minority children 
soften will not show the same Sehavibr patterns as do white children. 
Teachers expect children to demonstrate their giftedness in ways valued 
by the major culture. They rarely identify minority children as having 
extraordinary acaSemic talent. When the Vanguard Program, at that 
time implemented in two schools, was incorporated into the HISb Magnet 
Program^ hew ways of looking at talent and giftedness had to be 

o _ _ _ 

developed which were appropriate to the dbmihaht purpose of the 
Magnet Schools Program— the development of integrated education-as 
well as cbrigrueht with maintenance of program integrity and 
quality i In effect, the criteria for recruitment of students were 
broadened and became more complex; cultural standards for measuring 
intellectual talent had to be considered in addition to the more con- 
ventional test scores. 



After'much deliberation^ if was determined that Vanguard students 

should be recruited from varied ethnic and eobribmic backgrounds arid 

i 

demonstrate outstanding ability in two or more of the following areas: 

1. intellectual ability 

2. creative thinking 

3» leadership potential 

While test scores still form an important part of the assessment arid 
recruitment process , arid while students below a certain achievement 
level cannot qualify for admission to the program, a variety of data 
sources are used for identification of students^ including san^jles 
of writing^ and parerit arid teacher recommendations i The more flex- 
ible criteria allow students from different ethriic groups to compete 
among themselves for places in the program, father than against 
students who might be more advaritaged educationally or economicaixy. ^ 
The criteria also allow for the admission of. students who might be 
extremely gifted iri some areas, but who need extra help in others. The 
philosophy of the program states that ''By providing a stimulating and 
challenging environment which ericotafages^ ihteractibri with peers / the 
prog: am will develop the student's basic skills, creativity, * and in- 
tellectual abilities. Through specialized guidance programs and in- 
struct ion designed specifically to meet the needs of gifted studerits^ 
the program will provide opportunities for the student . to value him- 
self as a productive person with positive goals." 

With continuous and successful participation iri the program, the 
students ?^ill be able to ideritify ari area of interest, define the 



prbbiem, conduct research using the appropriate skills and techniques > 
and report the findings. They will be able to identify personal goals 
and opportunities for contributions to a field of interest and work 
effectively to realize their objectives," ^ 

To achieve these- goals, the classroom ehvlrbriment will emphasize 
interdisciplinary approauhes to basic reading, writing, mathematics, 
research/ group and reasbhihg skills. Opportunities will be provid- 
ed for every student to explore areas of interest in depth. Through 
the use of individualized instruction, each student is encouraged in 
the continual development of his unique potential i Each student will 
progress at a rate not to be impeded or unnecessarily accelerated by 
the developmental rate of other students i With stress on the heed for 



self-discipline / the student . .will be challenged by innovative and 

2 

creative motivation techniques." 

_ 

Since 1975, the Vanguard Program has been expanded to eleven 

campuses: six elementary schools, three middle or junior high schools, 

a senior high school, and a new campus ^ which integrates an elementary 

Vanguard program with a group of hearing-impaired students. Each 

Vanguard component is designed as a Schbbl-Within-A-Schodl , such that 

while each Vanguard campus has a Magnet c6-brdinator respbnsible for 

% 

the operation of its activities, building principals also exercise a 

_ _ _ 

strong* effect on impiementatlon of individual jprbgrams. The various 

prbgrams are dispersed throughout the Dis^c^ to facilitate service 

to all Houston students; care alsb is taken to locate programs in 

minority areas to facilitate the enrollment bf Black and Hispanic 



2 

"The Vanguard Philosophy" 
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students, llie program has proved to be popular; notwithstanding the 

difficulty in recruiting and transporting adequate numbers of minority 

_ 0 _ 

students for certain programs on the fringes ot the Houston area, wait- 
ing lists exist for most of the campuses. 

Evaluation of the Vanguard Program had followed the design for 
ail other Magnet programs. Each campus established objectives for 
their own program; yearly audits were made by the District's 
evaluation staff to assess compliance with these objectives and also 
with Court-mandated requirements for trahspbrtatibri of 'students, 
ethnic distribution of the student body, and ethnic cbmpbsition of the 
faculty. As a cbhsequehce bf tha dispersed nature of the program, the 
variance introduced by. building principals ? and the bpp6rtunity to 
est^lish objectives unique to each campus, considerable variatibn in 
* ' the programs existed. A desire fbr a ttidre comprehensive and descrip- 

tive form of evaluation was generated by the ^ need by the program 
directbr tb Icribw the extent to which the program had a separate identi- 
ty, or whether it consisted in practice of eleven distinct programs. 

A request was made to the District's evaluation department tb 
•^address the following questibns? 

1. Of what did the Vanguard program really consist? 

2. *rb what extent wer^ the individual campus programs congruent? 

3. Was the Vanguard Phiibsbphy-cited earlier-^really in evidence 
in the program? 

4. What impact was the prbgram having on studente? 

-6- 
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6 Si How did the. program cbnqpare with other programs for 'talented 
students? - 
6. Was the program really addressing gifted children? 
in order to answer these questions, it Was necessary to consider a 

number of cbristraintsi 

/ _ _ _ _ • . 

Purposes of the Evaluation 

t 

There were a number of reasons why the Vanguard staff wanted a 
change from past * evaluations of the program. First * the program 
director had designed overall program-wide objectivis which required 
comparisons among the individual campus prbframsi Second, while the 
program had grown considerably, detailed descriptive information on 
program implementation and practice had not been coUected. As a con- 
sequence, it was impossible to make any solid generalizations regarding 
Vanguard operations. There were constant pressures to open new 
schools, but there had been ho attempt to assess systematically the 
effect of the program on students. Most critical, however, were two 
factors. First* traditional means for assessing program effectiveness 
have involved collecting pre-^d pbsttest data to measure gains in 
pupil achievement. Such tests are* however, predicted upon the achieve- 
ment of normal children, they are relatively ineffective for measuring 
the gains of children who score at the extremes of norming tables — either 
very high or very low. In the case of Vanguard students, all had 

to score a minimum of one and one-half years above grade level to be 

3 ' . _ ■ 

considered for admission* Not only wcjuia^it have been unrealistic 

_^^^=. \ 

3 

in fact, the achievement of virtually all Vanguard students is con- 
siderably higher than one and one-half years above grade level. 

-7- 
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to expect dramatic gains in percentile rankings, but regression 
effects could have given the spurious appearance that tfie program 
actually harmed the children. While Vangxikrd administrators did antici- 
pate improvement in ^est scores of the lowest students, the general 
goal was maintenance of already excellent levels of academic achieve- 
ment. 

Another factor was that academic^ achievement represented only one 

of the; program's objectives fat students i Achievement test scores 

alone could hot assess validly such goals as increases in creative 

thinking, the ability to work ^independently, and' seljj^warehess. In 

fact, it' is doubtful whether any sort of paper-and-pencii test couid 

serve as a valid means for examining such gains. 

« 

Second, the types of questions posed by the program staff were 
not readily amenable to methods of assessment used previously i Some 
required eliciting from Vanguard participahts--teachers, students, and 
parents~descriptibhs of their activities and the meanings they attri- 
buted to their actions. Comparativ^^halysis of a variety of types of 
data were needed; Crucial to the evaluation would be actual observa- 
tion of classroom activities in order to determine the degree of 
Gommonality among the campuses. None of these could be done in a short 
period of time, and the District's evaluation department had limited 
resources to devote to the Project. Thus, a modified form of ethno- 
graphid evaluation was proposed. 
The Use- of 'Ethnography in Ev a Au a^ion - 

Ethnographies are analytic descriptions or reconstructions of 
intact cultural scenes and groups (Spradiey and McCurdy, 1972 5 which 
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delineate the shared beliefs > practices^ artifacts, folk kribwiedge and 
behaviors of some group of people. The design of ethnographic studies 
mandates investigative strategies conducive to cultural reconstructions 
First, these strategies elicit data which are pheribmehblbgical.' That 
is, they represent tJ\e worl^ view of the participants being investi- 
gated. Seconds ethnographic research strategies are empirical and 
* naturalistic; They involve acquisition of firsthand, sensory accounts 
of phenomena as they occur in real world settings; Third, ethnographic 
research is holistic. Ethhbgraphers seek tb cbnstruct descriptions of 
total phenomena within their various contexts and to generate from 
these descriptibhs major variables which affect human behavior and be- 
liefs toward the phenomena. Finally > ethribgraphy is multi-modal; it 
en^ldys a variety of research technologies (Wilson, 1977). Such an 
approach seemed tb fit the requirements of the requested evaluation; 

Three types of data provided by ethnographic research strategies 
seemed bf particular utility in this project: 

A _ . 

1; Base Line Data: these include information about the human 

.and techhblbgical ; context of the research population and , program 

t - - - - ■ - - - 

settings They are necessary for identification bf ^.ocial^ psychoid 

gical, cultural, demographic, and physical features of thfe context; for 

assessing the impact bf the program; and for establishing parameters 

* 

which could be used for generalizations to and cbmparisbri with bther 
settings and populations. In addition, the institutional framework 
and its relationship with other, institutions must be ^^xamined for the 
types of influence it exercised upon implementation arid charige within 

-9- 
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the progr^ (cf., e.g., Apple and Kihif, 1977; Sharp and Gieen, 1975). 

- - - ♦ - ' 

: 2. Pro|a5SS Data? These refer to informatibh determining what has 
Secured ifi the course of a' curricular program or innovation. The way' 
in which the program, ahd^ in fact/ the evaluation, is handled by 
parUcipants provides valuable data for assessing the impact and 
succfess o€ a prograiti; ; 

3* «Vaiues Data: These refer .to informatibh abbiit the values of 
the participants, the ..program administrators, and the policy makers 
who financed the program. -.The value implications of what participants 
do in a prbgram-=whose values the intervention supports and whbse are 
neglected— may dramatically affect the manner in wfjich the program "is 
implemented and the degree to which it^' is disseminated (See Suchman,- 
1967). • ^ 

- ' I 

The characteristics of ethnographic rese,arch outlined above con- 

fc. _ _____ 

tribute to providing ^re integrated base line and prbcess data and to 

genefating more comprehensive parameters for values- data : than do con- 

^/ --- 

ventional evaluation designs. * Research designs based upon _ 

cbnbihatibns of data collection meibhods and analysis strategies provide 

> r \ 

more complete and complex data on phenomena than do unimodal research 

designs, and as such, are appropriate for complex, multi-modal programs ^ 

like Vafigxiard. They possess more credibility^ because they enhance the \ 

reiiabiiity and validity of the evaluation restil^s (Denzin, 1978). 

_ - ^ - ' - ' _ 

Ethnographic strategies can be used in curriculum and program 

evaluation in two ways: comprehensive aSbptibn of- the entire 

ethnographic process or strategic selectibh of a few data collection 
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techniques. The Choice between theise alternatives is informed by the 
objective of the research. If the §bal is a descriptive product 

_: 

intended to document shared beliefs, practices, aft if acts ^ environments, 
folk knowledge^ behaviors, and subtle patterns of interaction, then the 
appropriate choice is the development of ah ethnography of the entire 
program. Smith and Keith's analysis (1971) of the establishment of ah 
innovative elementary school and Wax's (1980) documentation of the pro- 
cess of desegregation in five public schools are ethnographies com- 
parable to traditional investigation conducted by anthropologists and 
sociologists i They offer implicit or explicit explanations to account 
foi^the patterns observed; Such ethnographies of brgahizatibhal or 
curricular innovations differ from community and tribal studies only in 
their focus; They are costly, requiring extensive financing and highly 
trained personnel. Results may be inaccessible for several years. 

Clearly, such a design was inappropriate for the policy makers and 

, _ J_ _. ^ 

program administrators in the school district. 

Aa alternative is a iSbre limited choice of ethnographic data 

collection techniques. This provides some baseline, process, and values 

data, and may also strengthen the validity of instruments developed for 

assessment. It also my be used in the initial stages of the project to 

develop categories for structuring the evaluation itself. For example. 

Hall and Loucks (1977) used limited non-participant bbservatibh to 

assess the validity of a teacher questionnaire designed to determine 

the extent to which ihstrqctbrs used educatibhal innovations they had 

been taught in their classrooms; ftppiicatibhs sudh as this have the 

a^tfantage bf reducing the required time and resources, while producing 
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results soon available to program admlhlstratbrs. 

In the case of this study, we have chosen the latter alternative 
and given limited resources/^ we have planned a three year project i 
Presently^ we are midway through the first year. The plan .utilizes a 
combinatidn of data collection techniques, including s continuation of 
existing audit procedures 1^ which cbinpliahce with program-wide arid 
Magnet School objectives, as well as individual campus objectives, can 
be mbriitbred; design bf iriitirlSmentatibn for and collection of baseline 
data oh program implementation and activities^^ihcludirig methods 
teachers use for assessing student progress; observatibh of classroom 
activities; and design of comparative studied. Data collection tech- 
niques proposed Include the fbllowihg: - 

1. Elicitation of participant constructs 

2i Interviews with key ihfbrmahts 

3i Mapping and ehumeratibh bf the physical and socia! setting 

4. " Non-participant observation 

5. Cbllectibh of archival material, recbrds' and dbcuments 

These strategies * and the manner in which they will be uscsd are discussed 
belbw. 

Data Cbllectibh Techniques 

Eliciting of participant constructs refers tb the ^ prbcess of 
determining the set of "agreed upons" which inform the world of each 
participant. These include the categories into which pebple plassify 
items in their physical wbriS; the values which they use tb assign 
meaning to what they do; the categories of \ knowledge they deem important; 
the canons o^ discrimihatibh they use tb sbrt' items ihtq categories; and 

-12- 
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the rules by Whidh they assi'jri relationships to phinomena ih^heir world 
mimbali, 1965). There are a variety of ways to determini how partici- 
pants in a study define their world. These include specific surveys, 
sorting and ranking procedures such as q-sort techniques, and procedures 
which require participants to enumerate all the menlbers of a particular 
category of things. These constructs, once delineated, can be used as a 
means for explaining . wh^ " 
they also can serve aS a basis for comparison with other mbans for 
defining and assigning value to similar or identical phenomena. 

In the case of the Vanguard p^5g^am, it was not clear whether or 
not the teachirs and administrators agreed on the general goals and 
activities established by the program directory it also was unclear 
Aether or not program participants shared a similar philosophy with each 
other. TO this end, the initial phases of the evaluation involved deter- 
mining what vanguard teachers thought was unique about the program and how 
they thought it shbuld-and dtd-operate. These data are to be used for 
comparisons among groups of Vanguard participants and as the basis for an 
observational instrument which Will compare what Vanguard participants 
say occurs in the program and what actually appears to an outside non- 
participant observer. ' 

Key informants are individuals who possess special knowledge, status, 
status, or communicative skills and who are willing to share that 
knowledge and skill with a researcher (Zelditch, 1962). They frequent- 
ly are xhosen because they have access—in time. Space, or perspective— 
to observati5hs denied the researcher. interviews with key informants 
.can be used is a means for elibiting participant constructs, for 
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generating historical data, for corroborating observations made by a 

researcher, and for sensitizing a researcher to specific dilemmas or 
critical issues which exist within the phenomenon under investigation* 
in the case of this project, informants include the program director^ a 
number of campus cb-ordinatbrs > and certain members of the administra- 
tive staff of the Districts Other key informants may be identified, 
as the project continues/ particularly as investigation of parent and 
commuftiliy, attitudes toward the pfbgram begins* 

Formal and irifbrmal mapping of a research site involves becoming 
acquainted with participants > recording demographic characteristics of 
the population, mapping the physical layouts and creating a des- 
cription of the general context of the program or innovation under 

considerations In this prbcess, census of participants can be taken, 

_ _ _ - 

key informants may be identified, and a map of the use of t^me and 

space can be generated* Much process data may be gleaned as to the way 
in which participants taik about a program; this in turn can be used to 
determine the extent to which programs are being implemented uniformly 
or at . all. Since the Vanguard program was implemented on eleven 
different campuses, and since the host schools varied in the degree to 
which they had adequate facilities for the program, ah early concern 
has been the extent to which simple jphysical variation would affect the 
atmosphere and implementation of the program^ In addition, consider- 
able latitude as to curriculum has been allowed, and programs have 
varied in the degree to which they have maintained stable, staffing 
patterns* Biese factors needed to b^ considered in assessing the way 
the program functioned* " v 

ft nuniber of strategies for collecting ethnographic data might be 

-14- 
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called "non-interactive- in that they allow a researcher to gather 
material with little or no interchange with the participants or sub- 
jects of a study. Among these are two chosen for this evaluation 
projects non-participant observation and the analysis of documents 
and archival materials*- and records i 

Pelto and Peltb (1978) treat ridn-participant observation as a 

... _ c- _ 

category separate from participant bbservatibh.v^ It involves merely 

watching what is going on and recording events on the sp6t# As a 

_____ . . _ .11^ 

con^ieteiy distinct category* hoh-participant observation exists only 

where interaction is viewed from hidden cameras and fecdrdfcrs or 

through one-way mirrors. When researchers are observing bh the scene > 

which is the case in almost all public school settings, they recessari- 

ly'will interact to some degree with teachers and pupils. However, to 

the extent that interference in ongoing classroom events can be avoided* 

it is desirable i Mbst of the Vanguard students aire adcustomed to 

visitors, such that disturbance can be expected tb be minimal; in 

c 

addition, evaluation staff will use relatively unobtrusive instruments 
fdr recording data. 

Three forms df ndn-participant observation are used cbmmbhly by 
ethnographers: stream-of-br havior chrdnicles, analysis of prdxemics 
and kinfesics, and interact ibn analysis. The first meritidried technique 
requires accurate minute-by-rainute accounting bf what a participant 
says and dbes and wbuld be excessively time-consuming for the District 
to implement on a large-scale basis, hziyever' some atten^t may be made 
tb validate ihstruinentation by sampling from a small grbup bf class- 
rooms. More generally, the latter two methdds will be usedi m 

r 
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prbxemics and kinesics are concerned > respectivaly, with* the 

.t 

social uses of space ahd^with bodily movement (see Birdv^tstell^ 1970); 
Hall, 1974 ji Preliminary investigations indicate that Vanguard class- 
rboms Utilize time, spatial arrangements, and movemeat differently than 
do regular classrooms; further Retailed investigation of these patterns 
appears warranted. In addition, ' interaction between students and 
teachers and among students themselves in Vanguard classrooms seems to 
differ from that found in regular classrooms, even within a single 
school. Comparative data among the separate Vanguard programs arid with 
other non-Vanguard classrooms will be sought for corroboration. 

Finally, base line data can be aupented greatly by the collection 
ind analysis of records, dddvtmentii and archival materials. These may 
include textbooks, curriculum guides, memos, enrollment records^ 



\ 



minutes of meetings, student records/ handbooks, newsletters, lesson 
plans, student products, diaries-, logs, and bulletin board materials. 
Much of this material already is used in the^ monitoring of program 
objectives. It will, however, serve as a means for establishing cora- 
paritive data aihohg t^ie types of activities offered in the different 
Vanguard jprpgrams, and in Vanguard as compare^ with other HISD programs 
for high achieving students i 

Evaluation Model 



The evaluation model, as it is presently envisioned, is sequential 
and developmental; the later phases will be shaped by results of 
earlier -phases. Year _ one will involve the development of instruments, 
for assessing the program; in years two and three we will initiate com- 
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parisohs of Vanguard with other programs for stadents with high 
achievement* 

Phase One - Development of an Objective Description of the HISb 
Vanguard Program ^ 

X _ 

A. Definition of the characteristics of ^the Vanguar.d programs, 
it wiii require the foil wing steps, some of which already 
have been accomplished i 

\. Meetings and interviews with Vanguard instructors , co- 
^ drdinatdrs/ and program personnel to generate a general 
set of descrrptbi^ for the Vanguard program,- 

2. Development of a list of characteristics thought unique 
to Vanguard programs i 

3. Development of a preliminary open-ended observational 

K _ - - - 

checklist to be useS in a sample of Vanguard classrooms. 

4. Based upon the abov'e data, development of a questionnaire 
for teacher feedback bri validity of observational check- 
list and rank ordering of key characteristics of Vanguard. 

- 5. Final production of a low inference classroom observation- 
al instrument based upon the descriptive data gathered in 
observations and feedback from Vanguard personnel. 

B. Monitoring of program procedures/ objectives/ and activities 
at program-wide and school levels. These include the follow- 

i. Student recruitment procedures and selection criteria 
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2. Characteristics of selected students 

. V • 

3. Teacher recruitment procedures and selection criteria 

4. instructional procedvifes Ussighmerits, expectations, 
grading,; group instructional behaviors^ etc.} 

5. Characteristics of Physical facilities and resources 

6. Management policies and procedures 

7. Parental arid community involvement 

8. Student progress 
a* academic 

b. basic skills 

c. advanced and supplementary skills i.e., individual 
projects 

d. social 

9. Provision of appropriate inservice for teachers of gifted 
and talented students 

Phase Two - Expectancy Analysis 

fti Design samples of students , parents, teachers, administrators, 
cfeiiimunity peersons. Vanguard drop-outs, and qualified students 
hot participating in the Progirami 

Bi Construct interviews for abbv^ samples eliciting ideal and 
actual characteristics of the Vanguard program, reasons for 
possible discref)ahcies bet philosophy and practice^ and 

respondents reapiohs to the program in genefrals This pro- 
cedtiire will be developed first by utilizing open-ended one-to- 

ErIc ^. \ . 20 



one interviews with a small sainple from each of the grovips* 

From these results, questionnaires will be developed for 

distribution to larger samples. 

2 _ • 
Phase Three - Assessment of Vanguard Impact 

" [ I 

A. Iiitirature Review 

The characteristics and effects of enrichment programs of other 
school districts with which the HISD program might be compared 
will be deterfflined. The literature will be reviewed and a re- 
presentative sample of school districts contacted for ihfprma-- 

s- 

tion on enrichment programs in their districts. - ■ 

B. Con«)arisori of programs aSong individual Vanguard campuses 
Control, Group Cbmparisoh 

The last phase of the evaluation procedures will be to contrast 
the' characteristics of the HISD Vanguard programs to other 
programs With similar student populations but different 
curricular approaches. Osing the instruir.erits developed earlier 
in the project:, data from other HISD programs (e^g., Magnet, 
Major Works classes or regular classes) will be collected . 
Such a step will necessitate the cbllectibh of the samk type of 
data oh these other HISD programs as that collected oh the 

; ■ ; 1 

Vanguard progretms* ' 
Phase 1-A is to be conducted by the end of the 1980-1981 school year; 
i-B is part of the ongoing monitbring or auditing procedures carried 
out annually for ail Magnet Schools. Phase two wilt begin in the summer 
^5811 interviews will take place early in the 1981-1982 school year. 



EKLC 



Phase three will begin in 1981-1982 and is scheduled to contintae for the 
remaining two years of the evaluation project. Described baldv? are the 
evaluation procedures which have been implemented to date: 

1. Monitoring of program objectives by collection and analysis of 
records y documents, and archives 

2. Eiicitatibh of the teacher and administrator constructs by 
which the program is defined 

3. Initial mapping and development of baseline data 
Hbnitdrinq and todit procedures 

The campus level objectives are audited daring the school year to 
iHbnitor both progress being made for each of the campus-level program- 
wide objectives and compliance with the accompanying action plan, which 
stipulates special activities whiih v?ill be impiemehted in order for the 
objectives to be achieved. The campus-level objectives and accompany- 
ing plan, developed at the beginning of the school year, are coordin- 
ated with the. program-wide objectives and action plan. The program-wide 
objectives address the following, areas: idehtif icattion/selection of 
students, pareht/community involvement, academic achievement, and com- 
pletion of individual projects. Wie identification and selection 
objective specifies the nunSer of students to be tecruited into the 
program by the beginning of the 1981-82 school year. The second 
objective, parent and cbnimuhity involvement, seeks to increase the 
level of participation from these two groups. The purpose of the 
acadeinic achievement objective is to maintain the students' levels of 
achievement while they are in the program. The fourth 5bjective, 
cbn^letion of individual projects^ is designed to reflect the unique 
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features of the Vanguard programs The objectives and accompanying 
action steps at the cait^^us level were written to reflect the program- 
matic differences at each campus, e.g. differences in grade levels, 

___»v___ 

enrollment figures, arid particular programmatic emphases i 

Audits to monitor the objectives and action plan at each of the 
campuses occur twice during the school year* During a typical audit* 
site visits by the Research Assistant are made to obtain information on 
the status of each of the objectives and the degree of in^lemehtat.ion 
for each of the action bteps scheduled to occur * the type of informa- 
tion used to deterSine the status bh ah objective depends on the out- 
comes specified for iti For example, if the level of parental involve- 
ment is to increase as measured by the number of contacts made with 
parents during the school year, the hamber of contacts with parents at 
the time of the audit is examined^ implementation of the accbmpahyihg 
action plan is monitored by exaitdhihg previously agreed upon document- 
ation which demonstrates that the action steps in the plan were 
completed as scheduled^ Examples of action steps and accompany xhg 
documentation for a parental ihvblvertiexit objective include: 

Action "steps Dbcumehtatibh 

1. Organize a parent advisory li ttst of participating 
committee parents 

2. Hold meetings of special 2* Attehdauce lists 
intarest to Vanguard parents. 

3. Plan a variety of: interesting 3i iist of functions held 
functions which will appeal the school year 

to parents 

At the end of the year, a final audit is conducted at each of the 
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campuses to determine the extent to which each of the objectives is 
achieved, fis in the case of determining the status of objectives 
during the year, the nature of the ihfbrmatioif used to determine the 
degree to which objectives have been achieved depends on the specified 
, 'outcomes* 

Elicltat^ on of Partic t p ant Constructs 

c 

ftt the beginning of the school' year, two meetings were conducted to 
obtain input relevant to tfie Vanguard evaluation design. The first 
meeting included Vanguard administrative personnel and members of the 
district research departmerit. The second meeting included teachers and 
administrators associated with the Vanguard program. The meetings were 
designed to generate a general set of descriptors which Vanguard staff 
apply to program characteristcs and their activities. We began with the 
general' question: "What is Vanguard?" 

At the first meeting, discussion topics covered the assessment of 
gifted and talented student performance, the nature of instruction for 
gifted and, talented students, types of activities actually occurring 

• r - 

in the program, and potential .criteria by which the Vanguard program 
might be' evatuated* The following areas for assessing gifted and 
talented students were suggested by the participants: 
a. Traditional academic subject areas 

bi Process skills such as synthesizing, research, problem 

solving, and decision making 
c. Creativity iridluding divergent thinking^ fluency, elabbra- 

tton; originality^ and flexibility. 
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a. Sodialization as indicated by iiif-respohsibility and levels 
of maturity* ; 

Teachers and administrators characterized the nature of instruction for 
gifted and talented students as possessing a higher level of cognitive 
functioning compared to traditional teaching methods. It. emphasized 
less memorization and drill arid a greater degree of studerit self- 
reliarice. ^ey cited as illustrative activities occurring at the 
individual campuses, including foreign language courses, creative 
writing, individaal student projects, arid -science workshops at both 
the elementary arid secondary levels. In addition to sfandardized 
tests they suggested frhe following measures to evaluate the Vanguard 
program: teacher ratings of studerit behavior; an inventory of studerit 

b 

Isehaviors perceived by Vanguard itaff to exemplify creativity, intel- 
lectual ability and leadership; and sur^^eys of teachers' arid studerits! 
attitude Si 

ftt the second ineetirig. Vanguard ^teachers arid iristructional 
coordinators were askcS to provide descriptions of events and activi- 
ties which actually occurred in their classrooms as well as descrip- 
tions of the physical features of the classro5ms. " Events iricluded 

■ . - _ _ _ _ _* *<. 

types of student-teacher and studerit-student interactions. ClasSrcfoin 
activities" covered group activities, delivery of iristruction, use of 
curriculum materials, and iridividual student work. Descriptioris of 
the physical features included seatirig arrangements, spicial interest 
centers* arid rbbms fez specific fun^ctions. 



with Key infomahts ' * . 

Several informal interviews have been conducted with the prdgram 

w^irector^ who provided initial categories by which bur initial meetings 

were striicturea. Vanguard campus coordinators have been interviewed 

both formally and .informally as part of the audit procedure. These 

interviews have served to structure cbllectibri of base line datai 

» 

~„ . '' ' • 

Mapping and Enumeration > o£— the- ^hysi c al an d-Sociar Setting and Non^ 
participant Observation 

Mapping and enumeration of the physical and social settings in 
Vanguard classrooms has begun with preliminary bbservations bf 
elementary classrooms ^ The. descriptions include seating arrangements, 
tbpics discussed during the observations, and dehbtatibh bf students; 

<i ' " _ _ _ _ . 

: who made contributibhs during the class discussions, Pian^ for future 

bljservations include coliectihg more infbrmatibh in these areas^ the 

> -V - - 

proximity of Vanguard classrooms at each campus, types of materials 

used, and categories of student-teacher arid student-student interact 

tibhs. ^e data' gathered from the these preliminary bbseryatibns will 

be used to develop a low inference data collection instrument which 

Would be used by S nxinber bf observers ^fter a short training period* 
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